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METHOD OF INTERPOLATING COLOR PIXEL SIGNALS 
FROM A SUBSAMPLE COLOR IMAGE 



BACKGROUND 



This disclosure is related to interpolating pixel signal values for an Image and. in particular, 
interpolating color pixel signal values from a subsampled color Image. 
Background Information 

As is well-known, In a variety of circumstances, It Is desirable to perform color Interpolation. 

io 

!□ For example, for a camera or other imager that has the eapability of creating a color Image, typically it 
is too expensive to have the capability to capture three independent color pixel signal values for each 
pixel location of an image. Therefore, more typically, a subsampled color Image is captured and then 
the missing eolor pixel signal values are computed using techniques of color Interpolation, Most of the 
existing sirtipie Color interpolation techniques typically do not produce high-quality color images. 
Therefore, a need continues to exist for color interpolation capable of producing quality color Images 
from a subsampled color image. 
SUMMARY 

Briefly, in accordance with on© embodiment of the invention, a method of Interpolating color 
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pixel sfgdals from @ eubdampled eolor image In^ludds the fdllowtng. For ^ particular pixel i&dation in a 
subsampled color image, relative changes in a particular color pixel signal value are compared for two 
mutually orthogonal directions across the particular pixel location using pixel signal values Immediately 
adjacent to the particular pixel location. A color signal value for that particular pixel location for a color 
plane other than the color plane of the pixel signal value in the subsampled color Image of that location 
is computed by relatively weigh the pixel signal values, the relative weights depending, at least in part, 
on the relative change of pixel signal value level In a particular direction. 

p 

'S BRIEF DESCRIPTION OF THE DRAWINGS 

! ' 

\^ The subject matter regarded as the invention is particularly pointed out and distinctly claimed in 

i 

\^ the concluding portion of the specification. The invention, however, both as to organization and 

i — 

method of operation, together with objects, features, and advantages thereof, may best be understood 

H 

iS by reference to the following detailed description when read with the accompanying drawings In which: 
FIG. 1 is a schematic diagram Illustrating a subsampled color Image, such as a Bayer pattern 
color mage; 

FIG. 2 is a schematic diagram illustrating a portion of a subsampled color image in which the 
color pixel signal values are to be employed by an embodiment of a method In accordance with 
the present invention; 

FIG. 3 is a schematic diagram illustrating a portion of a subsampled color image in which the 
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color pixel signal values are to be employed by an embodiment of a method in accordanoe with 
the present invention; 

FiS. 4 is a sehematld diagram itlustrating a pdftbn of a dubddmpisd 06l6r Im^go In whbh th<» 
color pixel signal values are to be employed by an embodiment of a method in accordance with 
the present Invention. 
DETAILED DESCRIPTION 

In the following detailed description, numerous specific details are set forth in order to provide a 

thorough understanding of the invention. However, it will be understood by those skilled In the art that 
the present Invention may be practiced without these specific details. In other instances, well-known 
methods, procedures, components and circuits have not been described in detail so as not to obscure 
the present Ihveiition. 



13 



As previously indicated, in a variety of circumstances it may be desirable to have a capability to 
interpolate color pixel signals from a subsampled color image In order to recover the missing color 
components In a particular pixel location. Fig. 1 1s a schematic diagram Illustrating a Bayer pattern 
color image, which is one example of a subsampled color Image, although the invention is not limited in 
scope in this respect. For example, many other well-known subsampled color images may be 
employed as will provide satisfactory results. As previously indicated, unfortunately, many color 
interpolation techniques typically do not produce high-qualfty color images because the techniques 
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employed typically do not take into account, or at least reasonably correctly take into account, how the 
human eye perceives color. For example, a typical color interpolation technique may include averaging 
the pixels adjacent to a particular pixel location in which It was desired to interpolate the color signal 
value for those colors not included in that pixel location of a subsampled color image. However, 
typicallyi simply employing a pure average doe$ not produce high quality results because it blurs the 
edge Information and prdduees color ailaaing problem, for ek^mple. 

In this particular embodiment of a method of interpolating color pixel signals from a 

13 

subsampled color image in accordance wfth the present Invention, the color Interpolation is 

12 sccompliehod by comparing relative changes in mutually orthogonal directions acrods the particular 

b 

'•'^ pixel location using pixel signal values immediately adjacent to that particular pixel location. It is 

|~ assumed that if a significant change in a particular color pixel signal value occurs, that change is not as 

™ representative of the signal values at the particular pixel location as are the signal values from the 



rnutually orthogonal directtoh. F^or e^mple. It may be that in the Image, an edge Is being encountered. 
Therefore, in computing the color signal value for that particular pixel location, a direction that indicates 
less change in signal value is weighed more heavily, in this particular embodiment. It is believed that 
this is more representative of how the human eye perceives color because the human vision system is 
more responsive to the relative change of color in the neighboring pixels than to the color Information in 
the pixel itself, in addition, as explained in more detail hereinafter, in computing some pixel signal 
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values for particular odors, natural logarlthme are omploysd beoauee it la believed that natural 
logarithms capture how the human aye perceives relative color. 

Fig. 2 Is dchematic diagram tlluatratlng a three pUel-by-three pixel portion of the Bayer pattern 
ot Fig. 1 . As illustrated, the center pixel location comprises a red pixel signal value. Therefore, for 
this particular pixel location, it Is desired to interpolate the missing color signal values, blue and green, 
for that pixel location from the surrounding pixel signal values. For example, a green pixel signal 
value may be estimated as follows. As illustrated in Rg, 1 , the immediately adjacent pixel lecfttions in 
the horizbr^tal and veriicdl dire6tbns comprise green pixel signal values. Therefore, these shall be 
employed to estimate the green pixel signal value for this particular pixel location. Rrst, the relative 
change in the green pixel signal values for the horizontal direction and the vertical direction across this 
particular pixel location Is computed and compared. This Is accomplished using the following 
equations- 

ChOr = Q m.rv^l - G m.n-1 I 
' ever n GrrHl.n - G m-vn: 

If the relative change In the vertical direction Is greater than the relative change in the horizontal 
direction, the relative change being relative to the magnitude of the values computed above, then the 
values In the horizontal direction, that is, in this embodiment, the green pixel signal values that are 
the immediately adjacent pixel signal values in the horizontal direction, are weighed more heavily. In 
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this embodiment, the weight assigned to horizontal green pixel values have been chosen, based on 
experimentation, as 0.5, although the Invention Is not limited in scope in this respect It Is noted that 
other weights may be employed and provide satldfaetory results. At the same time the weights 
assigned to vertical neighboring green pixel signal values have been chosen as 0.1 , although the 
invention is not limited in scope in this respect-On the basis of the above discussion the missing green 
pixel signal values In this particular pixel location Is estimated as 

G rt^n = [ 0.5 • (G + G m.rwO + 0.1 * (Q m.i,n + G m+i.n) ] / (O.S + 0.5 + 0.1 + 0.1); or 
Q « 0.41667 * (G m.n.i t G m.^i) + 0.08333 ' (Q m-i.n + G m^i.n); 
^ However, if the relative change In the horizontal direction is greater than the relative change in the 
vertical dlraotlon, in terms of pixel signal level for th@ green pixel signal values, then a reverse 
approach is ehiployed. More particularly, the vertical green pixel signal values that are immediately 
adjacent to the red pixel signal value, in this particular embodiment, are weighed more heavily. In 
particular, the green pixel signal value in this particular pixel signal location is estimated as follows. 

0.08333 " (G ) + 0.41667 • (Q m-i.n + G m+l.n); 

It is noted that the form of this equation is similar to the fonn above, except that the vertical and 
horizontal pixel signal values that are immediately adjacent to the red pixel signal value have been 
Interchanged. Finally, if the two relative changes are equal, or substantially equal, then a simple 
average of the four green pixel signal values that are immediately adjacent to the red pixel signal 
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value are averaged, for this embodiment, In accordance with the following equation* 

G m.n = 0.25 * (Q m.n-l + G m,n+i + G m-1,n + G m+l.n )l 

Therefore, in order to compute the signal value for the green color plane, where the particular pixel 
location has a pixel signal value in the red color plane, the pixel signal values immediately adjacent 
TO that pixel lodation Irt th© qmn 6d\or plahs m oQffipmQ. As «hall bd dederlbdCI in mm 
below, it is not always the case that the color plane being computed con-esponds to the particular color 
of the pixel signal values that are compared, although It Is true in this embodiment for the situation just 
described. 

It is noted that where the particular pixel location has a blue pixel signal value In the Bayer 
pattern shown in Rg. 1 , the structure of a three pixel-by-three pixel portion of the sub-sampled color 
image is similar in arrangement to that illustrated in Fig. 2, except the center pixel contains the blue 
Information instead of red. Therefore, for a pixel location containing the blue signal value, the 
nnissing grden pixel signal value for that pixel location may be estimated by using the same technique 
previously described. 

Fig. 3 is schematic diagram of another portion of the Bayer pattern of Fig. 1 . This three pixel- 
by-three pixel portion is essentially the same as that shown in Fig. 2, except that pixel signal values 
in the respective corners of the three-by-three anray have also been provided. For this portion, it Is 
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desirable to interpolate a blue pixel signal value for the location In which a red pixel signal value is 
provided. However, as illustrated in Fig. 3, blue pixel signal values immediately adjacent to the red 
pixel signal value are provided in the corners. Therefore, in contrast with the approach Just described, 
the two mutually orthogonal directions to be compared comprise the main diagonal and the secondary 
diagonal of the array of pixels- However, instead of comparing the relative change in pixel signal 
value level for the blue pixel signal values directly, the relative change in hue across these two 
diagonals shall be compared. Hue, In this context, for rod and blue, refers to the relative dgnal value 

^ of a blue or red pixel signal value to a green pixel signal value. The reason this approach is employed 

n 

i| is because it is believed that hue information more accurately reflects how the human eye perceives 

o 

changes in color. Furthermore, a comparison for red and blue pixel signal values is made with 
green pixel signal values because, in RGB color space format, the green plane includes a relatively 
^0 larger amount of luminosity signal information compared to the red and blue and hue of the red or the 
blue is estimated by comparing with the green value. Therefore, hue for the four blue pixel signal 
values is computed by comparing the blue pixel signal value in that location with the green pixel signal 
value in that location. It should be noted that the missing green pixel value has already been 
recovered via the previous technique. Therefore, the following equations apply. 

hue_nw = B m.i,n-i - G m-i.n-i; 
hue_SW = Bm+i.n.1 -Grni-vn-i; 
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hue_ne = B m-i^n+i • G m-i.m.i; 

Then, the hue along the main diagonal and the aecondary diagonal are computed as follows. 

hue_md = hue_nw - hu©_se; 
hue_sd = hue_ne - hue_sw; 

As discussed above, if the hue in any particular direction changes mora significantly, then the 

hue in the orthogonal direction la employed to estimatd the blue pixel signal value In a particular pixel 

_ location. More accurately, the hue in that direction in which there is less change in hue is 

weighed more heavily than in the direction in which change In hue Is more .This strategy is employed 

to estimate the hue in that particular pixel location- However, irom the hue. the blue pixel signal value 

= . may be estimated as follows. 

p 

P log(B m.n) - IOg(G m.n) + 0.41667 * (log(B m.i.n.i/ G m.1.n.l) +log(B m.i,n.i/ G r.^i.r>^i)) + 
0.08333 •(log(B^.i.^,/G m-i,n+i) + log(B n>»1,n-i/. G m+1.rvl))J 



o 
la 



= 0 



Likewise, a similar equation is employed for the opposing diagonal, except that the signal values 
along the diagonals are interchanged, providing the following equation. 

log(B „,.„) = log(G m,n) + 0.08333 ' (log(B n,.,,n.i/ G m.i.n.i) + log(B „v.1.n*l/ G m*1.n+l)) + 
0.41 667 •(log(Bn,-,.n*i/G ) + IOg{B n>^i,n.i/Gr»HT,rvl)) 

Likewise, as mentioned previously, the natural logarithm is employed in this particular 
embodiment because it is believed that It reflects or captures how the human eye perceives relative 
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color. It 13 notod th@t interpolation of a rod pixdl signal value for a particular pixel location 

having a blue pixel signal value may be obtained using a slnnilar approach except that in the above 

equation the blue pixel signal values are interchanged with red pixel signal values. This is because. 

Ifluatrated in Fig* 1 , where the particular ptx«) Ideation has a blue pixel signal value, the pixel signal 
v^iudd in ths fdur amm m red piyd signal vaiudd, therefore, pfdvtdhgj ih^ Sams dtPu^u^s dt trfty 

layout as illustrated in Fig- 3. 

Finally, a blue pixel signal value may be Interpolated where a green pixel signal value Is 
provided in the particular pixel location of the subsampled Image, as illustrated in Fig. 4, for example. 
The approach is similar to that described in connection to that described with Rg. 3, except that, 
instead of employing the pixel signal values along the diagonals, the horizontal and vertical pixel 
signal values are employed using either horizontal or vertical pixel signal value directly available from 
the subsampled color image. Th© other pixel value In the orthogonal direction which Is used for 
interpolation is obtained by employing the previous technique. Likewise, the blue pixel signal values in 
the horizontal locations immediately adjacent the green pixel signal values in Fig. 4 are interpolated 
using the approach just described above in connection with Rg. 3. Therefore, the following equations 
are employed. 



u 
-E 



3~J 



hue_n = B m-i.n - G nv1,n • 
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hue^e = B n,.,>*l - G m,n+l . 
hue_W = B n,.n-i - G m.rv, . 
hue_S o B m+1,n - G n,*i.n. 

hhor a hu©_e - hue_w. 
hver = hue_n - hue^s. 

Hi I hhorj < |hver| ) then 

log(B m,n) = log(G „,,„) + 0.41667 • (log(B m.^.,/ G m-n-i) + log(B mW G „.r^^)) + 
0.08333 * ( log(B m-1 V Q m-i,n) + l0g(B m+l V G> m+i.n))- 

e/se 

y log(B m.n) = log(G ^.n) + 0.08333 * (log(B ) + log(B 

;P 0.41 667 • (log(B V G „,.!,„) + log(B G 

!3 end/f 

iri 

is 

P A sirhliar approach may be employed to interpol&td thd rdd pixel signal value where the particular pixel 

6 s 

?□ location of the subaampled image includes a green pixel signal value, except that In the equations 
=:0 provided above, the blue pixel signal values are replaced by red pixel signal values, as follows 
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hU©„n = R mo.n - G m.i.n • 

hue_e = R ^.n+i - G . 

hue^W = R - G m,n.1 ■ 
hue_S - R m.l.n ' G m+l.n. 
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tf(|hhor|<|hver|)f/ien 

log{ft m.n) *= loo(Q m,n) + 0.41667 * Oog(R Q ^.m) + log(R ^.„+i/ Q „,,„,,)) + 
0.08333 • ( log(fl + log(R «))■ 

e/ge 

log(R m.n) - log(Q m,n) + 0.08333 • (IO0(R ,n.M/ G m.rvl) + l09(R -T.,n*i/ G m.n*l)) + 
0.41 667 * (log(R m-iV G m-i,n) + l08(R m*l.n/ G m+i.n)) 

y 

S encf/f 

1 Li "" 

□ 

]^ it will, of coures, bo appreciated that the Invention Is not rdstrictdd in scope to a particular 

b 

; embodiment or implementation. For example, the foregoing approach, as one example of an approach 

i c=;. 
= =J 
i ssk; 

M in accordance with the invention, may be implemented in hardware, in software. In firmware, and any 

i— - 

p combination thereof. Aflain, Intended merely aa examples that do not limit the scope of the Invention, 
an embodiment may comprise an imager including hardware, such as integrated circuit chips, that 
implement the foregoing. Alternatively, the imager may be coupled to a computing platform that 
includes software capable of implementing the foregoing. Likewise, a digital camera coupled to a 
desktop personal computer, for example, may implement an embodiment Furthermore, these 
implementations in hardware and software may, of course, deviate from the foregoing and still be 
within the scope of the present invention. 
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For embodiments that are at least partially implemented in software, such as, for example, the 
previously described embodiment, such software may reside on a storage medium, such as, for 
example* random access memory, a CD ROM, a floppy diek, or a hard drive, such that instructions are 
stored, which, when executed, such as by a computing platfonri, such as a PC or other computing 
device, the system is capable of executing the instructions to result in the interpolation of color pixel 
signal values from a subsampled Image. Ukewl&e. such software may reside in firmware also, such as 

f in flash memory or EEPROM, for example. 

J 

While certain features of the invention have been Illustrated and described herein, many 



is ' ■ ' 

M modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, 

H 

therefore, to be understood that the appended claims are Intended to cover all such modifloations and 

p 
p 

If chahgefe as f^ll Withih the true spirit of the Invehtlon. 

' is: 
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